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Abstract 

The main purpose of this study was to examine a polytomous IRT-based linking 
proeedure that adjusts for rater variations. Test scores from two administrations of a 
statewide reading assessment were used. An anchor set of Year 1 students’ constructed 
responses were rescored by Year 2 raters. To adjust for year-to-year rater variation in 
IRT linking, a two-step approach was used. First, the Year 1 item parameters for the 
constructed response items were re-estimated using the ratings assigned by the Year 2 
raters. Through this recalibration process, the Year 1 item parameters for the constructed 
response items were adjusted for rater variation. Second, the Stocking-Lord equating 
method was used to place the Year 2 form on the Year 1 scale using all the common 
items between forms. 

This method was compared with two alternative methods: traditional IRT linking 
study that links the test forms using a) all the common items without the rater adjustment 
and b) the common selected response items. The success of the Stocking-Lord procedure 
was compared among the three methods. The differences in the test characteristic curves 
and the students’ scale score distributions produced by the three methods were also 
compared. 

Significant shifts in the parameters after rater adjustment were found for one 
(grade 8) of the three grades examined. The p-values and TCCs shifted across years 
when adjusted for rater effects. The impact of the parameter shifts and TCCs manifested 
in the changes in the proficiency classification before and after adjustment. However, a 
systematic bias might have been introduced in the equating process while trying to adjust 
for the rater variation through the equating process. Further studies are needed to address 
this problem in greater detail. Using only the common SR items seems to produce 
satisfactory results. Thus, in the case where it is not feasible to integrate rater adjustment 
in the equating process, using SR item anchors is a better approach than using the mixed- 
item anchors without adjusting for rater effect. 

The results of the study suggest that raters were not consistently more severe or 
more lenient between grades, but the resulting rater error (severity or leniency) affected 
the scores and thereby produced misleading results if not taken into account. 
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Introduction 

Constructed response items have beeome a standard part of edueational 
assessment. The inelusion of these items is motivated primarily by validity arguments. 
They are thought to be a more authentie evaluation of student’s eompetenee.. Several 
eritieal measurement issues related to the use of mixed item format tests have been 
investigated by measurement experts, e.g., Tate (1999, 2000); Wilson & Wang (1995). 
One of these issues in linking mixed item format tests is score eomparability aeross test 
administration years. One eommon equating design used in linking or equating tests 
from year to year is item response theory (IRT) sealing using a non-equivalent, eommon 
item equating design. The traditional linking method often applied to linking test forms 
administered in different years is an anehor test design. If the eonstructed response (CR) 
items are ineluded as part of the anehor set with selected response (SR) items, there may 
be the presence of year-to-year rater variation. The need to have the anehor item set 
inelude all elements of the test specifications suggests sueh inelusion. However, year-to- 
year rater variation might result in ehanges in the parameters of the CR items, making 
sueh inelusion inappropriate. 

Researeh on rater issues has shown that rater severity has signifieant impaet on 
student’s seores (Engelhard, 1992, Lunz, Writing, & Linaere, 1990, Patz, 1999) and 
therefore eannot be ignored. Wilson and Wang (1995) found that although inter-rater 
eorrelations were moderately high in their study, the range of rater severities was quite 
large, and that the rater effeets on the eonstrueted response items ean impaet student 
seores. Studies (Ito & Sykes, 1998) have shown that even if a test has a small number of 
eonstrueted response items, sueh as CRs only eontributing about 30% of total points 
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possible, a substantial amount of year to year rater fluetuation can yield relatively large 
score differences even after the test is equated through selected response items to an 
existing scale. Fitzpatrick et al (1998) found that the variation of rater severity changed 
every year for a statewide assessment. And that these variations could affect students’ 
proficiency classifications, indicating the need to adjust for rater effects during the 
equating process. Therefore, linking procedures that do not take rater severity into 
account may produce misleading results. 

Tate (1999) argued that raters differ in discrimination and severities over time and 
that applying the standard linking method through common CR items result in incorrect 
results since year to year changes in the anchor item parameters may be due to both to 
changes in the anchor item parameters and to changes in rater discrimination and 
severity. Although a relatively convenient option is to use only selected response (SR) 
items in the anchor sets, as the number of constructed response items increases in a test, 
the impact of rater severity or leniency on scores becomes greater. Tate (2000) has 
shown that linking based on SR anchor items can sometimes produce accurate results for 
a mixed item format test. However, such a linking method is only defensible if the 
selected response items and the constructed response items measure the same construct. 

Since the use of only SR items for linking may lead to serious linking bias under 
some conditions, it is suggested by some authors (Fitzpatrick et al, 1998; Wilson & 
Wang, 1995; Tate, 1999, 2000) that it is necessary to apply an equating methods to adjust 
for the rater effects. Tate (1999, 2000) adjusted for rater effects through IRT linking 
using simulated data. His procedure involves conducting a linking study in which a 
sample of anchor item papers for examinees from the prior year is inserted into the 
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scoring process for the current year. He further stipulated that the sample of prior year 
papers needs to be representative of all the papers and that the speeifie raters judging the 
anehor papers from the prior year must be the same raters that judge the current year’s 
papers for that item. An initial estimate of the item parameters were obtained for both 
years. Then the item parameters assoeiated with the CR anchor items on the prior year 
papers were re-estimated using the seores assigned by the eurrent year’s raters. These 
‘adjusted’ item parameters were then used in the final sealing for the eurrent year form. 
The final seale for the current year was obtained by eondueting a Stoeking & Lord (1983) 
equating proeedure using all the eommon items between the forms. Since the CR item 
parameters have been adjusted to refieet the rating standards for the eurrent year’s raters, 
eross year rater effeet have been held eonstant through the linking proeedure, the year-to- 
year ehange in ability was eorreetly refieeted in the students’ ability estimates. 

Wilson and Wang (1995) used a Random Coeffieient Multi-nominal Logit Model 
to model both the item difficulties and rater severities in the same model. In particular, 
the item parameters of the eonstrueted response items were deeomposed into linear 
eombinations of the item difficulties and rater severities. Three parallel test forms were 
administered to three different groups of examinees. There was one eommon SR item 
aeross the three forms. Two of the forms have additional SR items in eommon. Two of 
the forms have eommon raters. Common items and eommon raters were used to link the 
three forms together. The item parameters of the three forms were estimated 
simultaneously rather than separately. Rater effeet was removed using the simultaneous 
ealibration proeess. 
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Both Tate’s (1999, 2000) and Wilson and Wang (1995) methods attempt to adjust 
for rater effeet in the IRT linking process. Tate’s method is applicable to IRT models in 
general and most IRT calibration and equating software is sufficient. Wilson and Wang’s 
method is limited to the Rasch family of models and it requires specialized programs. 
Although Tate’s method shows a lot of promises, only the simulation data were used in 
the study. More research using real test data is needed. 

The purpose of this study is to evaluate the success of adjusting cross year rater 
variation on IRT linking using empirical data. An IRT linking similar to the method 
proposed by Tate (1999, 2000) was used to adjust for rater variation between years. This 
study differs from the Tate’s studies in several ways. In Tate’s simulation studies, using 
a two-parameter model and graded response model for data calibration, it was assumed 
that the Year 2 raters rescored a sample of anchor item papers from examinees in Year 1. 
This study uses real test data from a state assessment using a three-parameter model and 
the partial credit model. The entire set of anchor item papers from examines in Year 1 
was rescored in Year 2. Lastly, Tate assumed that specific raters who judged the prior 
year’s papers for the anchor item were the same raters who judged the current year’s 
papers for that item. This assumption is not likely to hold in live, large-scale testing 
situations. In this study, the raters that judged the prior year’s papers are not the same 
raters as those who judged the current year’s papers. However, training protocols, 
processes, and materials were the same between the two years. 

Method 

Data from a statewide reading assessment were used to investigate the effect of 
cross year rater variation on the year-to-year form equating. Year 1 and Year 2 forms 
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were eonstrueted to be parallel. The test eonsisted of 75, 93, and 91 SR items and 12, 14, 
and 16 CR items for grade 4, 6, and 8, respectively. All the CR items were scored on a 
three-point scale (0-2). The number of score points and the percentage of the total 
number of score points on the test attributed to the non-anchor items and the anchor items 
by item type are presented in Table 1 . The anchor item block consists of both the SR 
and the CR items. 

A three-parameter logistic model was applied to the selected response items 
(Lord, 1980) and two-parameter partial credit model was used for the constructed 
response items (Muraki, 1992.) Items within each grade were calibrated using PARDUX 
software program (Burket, 2002.) 

Table 1. The Number of Score Points by Item Type 

Non-Anchor Anchor 





SR 


CR 


SR 


CR 


Reading 










Grade 4 


65(75%) 


4(5%) 


10(11%) 


8(9%) 


Grade 6 


79(74%) 


4(4%) 


14(13%) 10(9%) 


Grade 8 


80(75%) 


6(6%) 


11(10%) 10(9%) 



The test was administered to approximately 40,000 students per grade. So that 
complete blocks of rater data were available, two samples, designated as Year 1 and Year 
2 were selected. The Year 1 stratified random sample consisted of about 1,200 students 
drawn from the population of test takers in the state. Similar sampling procedures were 
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used to obtain the Year 2 examinees of about 2,000. Eaeh rater seored only a portion of 
the CR items on a test. This is a frequently adopted strategy for redueing the halo effeet. 

To evaluate ehange in the ratings of the populations of raters between years 
requires eomparisons of the ratings obtained for the student papers for the Year 1 tests. 
Raters that judged the first year papers were not the same raters that judged the seeond 
year papers. However, the same training proeess and materials were used in training the 
raters in the two eonseeutive years. Responses to the eonstrueted response items for eaeh 
student, in the form of their original papers, were reseored by a representative sample of 
raters who seored the Year 2 operational examination. This sample of readers was 
retained towards the end of the seoring session to reseore the Year 1 student papers. 

These raters were not aware of the previous year’s seore. 

Raw-seore statisties for the eonstrueted response items are presented in Table 2. 
The differenees in raw seore means and standard deviations between years tend to be 
small. The exeeptions are item 99 in grade 6 and items 91, 92, and 93 in grade 8. Item 
91 in grade 8 showed the largest raw seore mean differenee between years. The 
differenee of 0.65 is espeeially large for a two-point item. The generally small 
differenees in means suggest that the H*- and 2"‘*-year raters, on average, awarded very 
similar seores. However, the year 2 raters tended to be more lenient than year 1 raters in 
grading the CR items. 

The pereentage of perfect, adjacent and discrepant agreement between years is 
presented in Table 3. The percent perfect agreement varied from 40% to 88%. Grade 8 
tended to have lower perfect agreement rates compared to grades 4 and 6. Again, item 99 
in grade 8 showed a very low agreement rate between the Year 1 and Year 2 raters. 
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Table 2 

Mean and Standard Deviation of Item Scores Assigned by and 2“** Year Raters 





































































































Table 3 

Across- Year Rater agreements 
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Analysis 

The polytomous IRT -based linking proeedure used in this study to adjust for rater 
variation in the linking proeess is summarized before. First, the Year 1 item parameters 
for the CR items were estimated using the ratings assigned by the Year 2 raters. Seeond, 
the Stocking-Lord (Stocking & Lord, 1983) equating method was used to place the Year 
2 test on the Year 1 scale using all the common items. Since the year-to-year rater 
variation has been accounted for in the first step, the changes in the constructed response 
anchor item parameters are reflective of changes in item difficulty only as opposed to a 
combination of rater variation and item difficulty. For ease of discussion, this procedure 
is referred to as mixed-item anchors with rater adjustment. 

Two alternative methods were also implemented for comparison purposes. One 
uses mixed-item anchors with no rater adjustment and the other uses SR items as anchor. 
The former is a simply a traditional IRT linking procedure using both the SR and the CR 
items as anchors. The latter uses only the SR items as anchor. In both of these two 
procedures, item parameters for the CR items were estimated using the ratings assigned 
by the Year 1 raters. For both of these two procedures. Year 2 item calibration was 
conducted. Then the Stocking-Lord equating method was used to place the Year 2 test on 
the Year 1 scale using the appropriate anchors. 

The mixed-item anchors with rater adjustment procedure tries to integrate rater 
adjustment into the equating process. Operationally, Year 1 item parameters for the CR 
items were re-estimated using the ratings assigned by the Year 2 raters. This was 
accomplished by replacing the ratings assigned by the Year 1 raters with those of the 
Year 2 raters for the same students. Note that item parameters for the SR items were 




also re-estimated in this proeess. Then the Stoeking-Lord (Stoeking & Lord, 1983) 
equating method was used to place the Year 2 test on the Year 1 scale using all the 
common items between forms. 
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Results and Discussions 

Summary statistics for equating Year 2 and Year 1 form are shown from Table 4 
to Table 6 for each grade. Table 4 showed that grade 4 students who were administered 
Year 2 form had a number correct score mean of 9.78 on the common SR and CR items. 
Students who were administered Year 1 form had a number correct score mean of 9.95 
and 9.98 (after rescore) on the same items. Thus, based on the combined SR and CR 
score. Year 2 students appear to be lower performing students than the Year 1 students. 
Similar conclusion would be made when comparing the number correct score mean 
between years using either the common SR items or the common CR items. Based on 
this finding, using mixed-item anchors, mixed-item anchors adjusted for rater variation, 
or SR item anchors should produce similar equating results. 

For grade 6, based on the common SR and CR items. Year 2 students appear to be 
slightly lower performing than the Year 1 students. However, the differences in the 
number correct score mean between years were very small. Similar conclusion would be 
drawn when comparing the common CR or the common SR means between years. 
Therefore, it is expected that Year 2 students’ performance will be very similar to those 
of the Year 1 students. Furthermore, Year 2 students’ scores might not be affected by the 
choice of the anchor set for equating. Using mixed-item anchors, mixed-item anchors 
adjusted for rater variation, or SR item anchors should produce similar results. 
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For grade 8, comparing the performance of Year 2 and Year 1 students using the 
number correct score mean between years were difficult because different conclusions 
would be reached depending on which anchor set was used in making the comparison. 
When the common CR item means were examined, Year 2 students appear to be slightly 
higher performing (mean=5.54) than the Year 1 students (5.23.) However, after the 
rescore. Year 2 students seem to be lower performing than the Year 1 students 
(mean=6.5 1 .) This finding is consistent with the across year rater variation reported 
earlier. Lastly, based on the common SR items. Year 2 students (mean=7.58) appear to 
be lower performing than the Year 1 students (mean=7.68.) Thus the Year 2 students’ 
performance on the rescored common CR items is more in line with those based on the 
SR items. Based on these findings. Year 2 students’ scores might be inflated if the 
common CR items were included in the anchor and that rater leniency was left 
uncorrected. Furthermore, using mixed-item anchors, mixed-item anchors adjusted for 
rater variation, or SR item anchors are likely to produce different equating results. 
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Table 4. Summary Statistics for Equating Year 2 and Year 1 Form for Grade 4 



Score 


Year 


Mean 


SD 


Skewness 


Kurtosis 














Total Score 


Year 2 


55.03 


19.35 


-0.47 


-0.77 




Year 1 


56.55 


18.38 


-0.45 


-0.73 




Year 1 Rescore 


56.42 


18.52 


-0.42 


-0.79 














SR+CR anchor 


Year 2 


9.78 


4.59 


-0.27 


-0.99 




Year 1 


9.95 


4.41 


-0.28 


-0.82 




Year 1 Rescore 


9.98 


4.49 


-0.25 


-0.9 














CR anchor 


Year 2 


3.8 


2.56 


-0.03 


-1.22 




Year 1 


3.77 


2.42 


-0.06 


-1.13 




Year 1 Rescore 


3.8 


2.52 


-0.02 


-1.21 














SE anchor 


Year 2 


5.97 


2.51 


-0.39 


-0.67 




Year 1 


6.18 


2.48 


-0.46 


-0.64 




Year 1 Rescore 


6.18 


2.48 


-0.46 


-0.64 



Table 5. Summary Statistics for Equating Year 2 and Year 1 Form for Grade 6 



Score 


Year 


Mean 


SD 


Skewness 


Kurtosis 














Total Score 


Year 2 


72.39 


20.95 


-0.92 


-0.26 




Year 1 


72.21 


20.34 


-0.66 


-0.23 




Y ear 1 Rescore 


72.31 


20.29 


-0.67 


-0.23 














SR-l-CR anchor 


Year 2 


15.38 


5.03 


-0.78 


0.1 




Year 1 


15.49 


5 


-0.62 


-0.17 




Year 1 Rescore 


15.52 


4.89 


-0.66 


-0.07 














CR anchor 


Year 2 


5.09 


2.46 


-0.33 


-0.59 




Year 1 


5.08 


2.52 


-0.18 


-0.73 




Year 1 Rescore 


5.1 


2.4 


-17 


-0.63 














SE anchor 


Year 2 


10.29 


3.02 


-0.98 


0.55 




Year 1 


10.41 


2.91 


-0.91 


0.43 




Year 1 Rescore 


10.41 


2.91 


-0.91 


0.43 
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Table 6. Summary Statistics for Equating Year 2 and Year 1 Form for Grade 8 



Score 


Year 


Mean 


SD 


Skewness 


Kurtosis 














Total Score 


Year 2 


77.46 


21.54 


-0.83 


-0.12 




Year 1 


77.4 


21.6 


-0.79 


-0.24 




Year 1 Rescore 


78.62 


21.52 


-0.83 


-0.16 














SR+CR anchor 


Year 2 


13.12 


4.66 


-0.6 


-0.37 




Year 1 


12.91 


4.56 


-0.43 


-0.56 




Year 1 Rescore 


14.2 


4.46 


-0.62 


-0.38 














CR anchor 


Year 2 


5.54 


2.4 


-0.4 


-0.46 




Year 1 


5.23 


2.43 


-0.13 


-0.59 




Year 1 Rescore 


6.51 


2.36 


-0.6 


-0.19 














SR anchor 


Year 2 


7.58 


2.73 


-0.66 


-0.47 




Year 1 


7.68 


2.6 


-0.64 


-0.41 




Year 1 Rescore 


7.68 


2.6 


-0.64 


-0.41 



To examine the effeet of integrating rater adjustment proeess in the Stoeky-Lord 
equating proeedure, the degree of alignment between the item diffieulty of the anehor 
item parameters and their aetual p-values was eompared between the two mixed item 
type linking methods. There are various different ways to evaluate the Stoeky-Lord 
equating proeedure. In this study, we ehose a whole item approaeh in order to summarize 
the overall impaet of applying rater adjustment to the equating results. Two items eould 
have somewhat different diserimination, diffieulty, and guessing parameters yet still on 
the whole perform in the same way. Consequently, it makes sense to look at them as a 
whole rather than to look at its individual item parameters. This approaeh refieets a 
whole item approaeh in evaluating the results of the Stoeking-Lord equating proeedure. 
The estimated p-values were eomputed using the anehor item parameters along with the 
ability distribution based on the anehor set. The aetual p-values were the elassieal p- 





15 



values. The p-values plots when both SR and CR were used as anehors are presented in 
Figure la to Figure 3b. The aetual p-values are displayed on the horizontal axis and the 
estimated p-values are displayed on the vertieal axis. A comparison of Figure la and 
Figure lb showed that the alignment between the actual and the estimated p-values got 
slightly worse after the rater adjustment. The correlation between the actual and the 
estimated p-values decreased from 0.95 to 0.92. Such decrease in alignment is evident by 
examining item 78, which is a CR item. Its estimated p-value was closer aligned with the 
actual p-value before the rater adjustment. This decrease in alignment between the actual 
and the estimated p-values for item 78 reflects effect of the rater adjustment in the overall 
equating results. Figure 2a and 2b showed that for grade 6 there was great alignment 
between the anchor and the equated parameters and there was no improvement in 
alignment with the rater adjustment. The correlation between the actual and the 
estimated p-values was unchanged. 

Figure 3a and Figure 3b showed that the overall alignment between the actual and 
the estimated p-values was relatively poor, with or without rater adjustment, as compared 
to grade 4 and 6. When the anchor and the equated parameters were produced in two 
consecutive years, the correlation between two sets of parameters is typically in the 90s. 

A closer examination of the plots reveals that the degree of misalignment is most severe 
for two of the CR items, item 91 and item 99. There was a significant improvement in 
how the actual and the estimated p-values aligned when the rater adjustment was applied. 
The correlation between the actual and the estimated p-values increased from 0.80 to 
0.86. The alignment for item 91 has improved, however, the alignment for item 99 
deteriorated after the rater adjustment. Furthermore, the estimated p-values after the rater 
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adjustment had been applied tend to under-estimate their actual p-values. This is evident 
by the disproportional number of estimated p-values for both SR and CR items fall below 
the diagonal line in Figure 3b. 

Figure 4 to Figure 6 present the actual and the estimated p-value plot for the three 
grades when the SR items were used to equate the forms. Since the CR items were not 
included in the anchor, the alignment between the actual and the estimated p-values 
improved, especially for grade 8. The correlation between the actual and the estimated p- 
values are in the high 90s for grade 6 and 8. 

Comparing Figure 6 and Figure 3 a, it is evident that the SR anchor items behavior 
similarly in the two plots. However, when the rater adjustment had been applied through 
the equating process (Figure 3b), the performance of the SR anchor items changed. 

Again, it appears that the rater adjustment that had been introduced in the equating 
process had a significant impact on the stability of the SR anchor items. 
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Figure la. Grade 4 P-value Plots without Rater Adjustment 




r=0.95 



Figure lb. Grade 4 P-value Plots with Rater Adjustment 




r=0.92 
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Figure 4. Grade 4 P- value Plots Using SR as Anchor 




r=.90 



Figure 5. Grade 6 P-value Plots Using SR as Anchor 




r=0.99 






Estimated P 
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Figure 6. Grade 8 P- value Plots Using SR as Anchor 




r=.98 
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Scoring of the Examinees 

The transformed item parameter estimates were used to eonvert the examinees’ 
scored responses to obtain the number-correet to scale score conversion. The ability 
estimates were plaeed on the scale seore with a multiplier of 20 and an additive of 200. 
The highest and the lowest obtainable seale seores were set to 100 and 300. To help 
visualize the results, the test characteristies eurves (TCC) based on the three anehor sets 
are presented in Figures 7 through 9. In these figures, the number correct scores were 
rescaled to pereent eorreet to faeilitate interpreting the results. The blue line showed the 
test eharaeteristie curve for Year 2 using mixed- item anehors without rater adjustment, 
the pink line showed the eharaeteristie eurve for Year 2 using mixed- item anchors with 
rater adjustment, and the green line showed the test eharaeteristie curve for Year 2 using 
SR items as anehors. The eorresponding standard error of measurement eurve was 
displayed at the bottom of each plot. For grade 4 and 6, three TCCs are almost 
indistinguishable. The standard errors of measurement eurves produced by the three 
proeedures are also similar. For grade 8, however, the differenees in the TCCs produeed 
by the three different proeedures were more pronouneed especially between the TCCs 
with and without rater adjustment. Although not shown on the plots, it was found that 
eighth graders who reeeived 50 percent of the maximum possible score on the test will be 
assigned a scale seore that is about eight points lower if the rater effeet had been adjusted. 
Furthermore, it appears that using SR anehors produeed more aeeurate results as 
compared to those using mixed-item anehors without rater adjustment. Turning to the 
standard error curves produced by the three proeedures; it seems that adjusting rater 
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variation had a negative impaet on the preeision of estimation for the higher performing 
students but had a positive impaet on the preeision of estimation for the lower performing 
students. 
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Figure 7. Test Characteristic and Standard Error Curves for Grade 4 




Figure 8. Test Characteristic and Standard Error Curves for Grade 6 
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Figure 9. Test Characteristic and Standard Error Curves for Grade 8 




Looking more closely at the effect of applying rater adjustment in the equating to 
the students’ scale score distributions, Tables 7 presents the percentage of Year 2 students 
who were classified as proficient for the three procedures as well as the percent of Year 1 
students being classified as proficient. It should be noted that both number-correct scale 
score and the item-pattern scale score were derived for the examinees. Since the 
differences between the two scale score distributions were very small across grades and 
comparisons and that item-pattern scoring is a more commonly used procedure than 
number-correct scoring, the former was used in this analysis. A somewhat arbitrary cut 
score of 200 was set to classify students into proficient or not proficient category. For 
grade 4, using the mixed-item type anchors with rater adjustment in the equating resulted 
in 2% less students being classified as proficient as compared to those without rater 
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adjustment. The differenee of 2% is relatively small and might be eonsidered random 
errors. Note that the result based on the SR anehors is exaetly the same as those based on 
mixed-item anehors with rater adjustment; therefore, it eould be argued that the three 
proeedures produeed the same results. 

There was virtually no differenee in the pereent of students being elassified as 
profieient for grade 6 among the three proeedures. Sueh findings are not surprising 
given the similarity in the TCCs across procedures. 

For grade 8, there was quite a significant decrease (14%) in the percent of 
students classified as proficient using mixed-item anchors adjusted for rater variation and 
a smaller but still sizable decrease (4%) in the percent of students classified as proficient 
using SR anchors. It seems that not accounting for rater variation can produce 
misleading results under the presence of significant rater variations. Table 8 to Table 10 
presents the summary statistics of the students’ scale score distributions for each grade by 
year and by procedure. Overall, the mean and the standard deviation are quite similar 
across the three procedures with the exception of grade 8. For grade 8, it seems that 
small differences in means among the three procedures might correspond to a sizable 
difference in the percent of students being classified as proficient. 



Table 7. Scale Score Distributions by Grade 

Percent Proficient (SS>200) 

Y ear 2 Mixed-Item 





Year 1 


Year 2 Mixed-Item 
Anchor 


Anchor With Rater 
Adjustment 


Year 2 SR Only 
Anchors 


Grade 4 


56.5% 


54.0% 


52.1% 


52.1% 


Grade 6 


52.0% 


51.5% 


51.5% 


51.5% 


Grade 8 


55.5% 


58.3% 


44.2% 


54.2% 
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Table 8 Grade 4 Reading Scale Score Summary Statistics 





Year 1 


Year 2 Mixed-Item 
Anchors 


Year 2 Mixed-Item 
Anchors With 
Rater Adjustment 


Year 2 SR Only 
Anchors 


Sample Size 


1019 


2129 


2129 


2129 


Mean 


201.11 


199.34 


198.10 


198.34 


Std. Error of the Mean 


0.800 


0.614 


0.600 


0.575 


Median 


202.9 


202.7 


201.4 


201.5 


Std. Deviation 


25.55 


28.32 


27.70 


26.52 


Confidence Interval (95%) 










Lower Bound 


199.54 


198.14 


196.92 


197.22 


Upper Bound 


202.69 


200.54 


199.28 


199.47 
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Table 9 Grade 6 Reading Scale Score Summary Statistics 





Year 1 


Year 2 Mixed-Item 
Anchors 


Year 2 Mixed-Item 
Anchors With 
Rater Adjustment 


Year 2 SR Only 
Anchors 


Sample Size 


1015 


2061 


2061 


2061 


Mean 


198.36 


197.40 


197.21 


196.81 


Std. Error of the Mean 


0.815 


0.562 


0.571 


0.575 


Median 


201.0 


200.5 


200.4 


200.0 


Std. Deviation 


25.97 


25.52 


25.92 


26.10 


Confidence Interval (95%) 










Lower Bound 


196.77 


196.29 


196.09 


195.68 


Upper Bound 


199.96 


198.50 


198.33 


197.94 
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Table 10 Grade 8 Reading Scale Score Summary Statistics 





Year 1 


Year 2 Mixed-Item 
Anchors 


Y ear 2 Mixed-Item 
Anchors With 
Rater Adjustment 


Year 2 SR Only 
Anchors 


Sample Size 


1001 


2156 


2156 


2156 


Mean 


202.16 


202.79 


194.45 


200.36 


Std. Error of the Mean 


0.876 


0.587 


0.598 


0.618 


Median 


203.3 


204.6 


196.3 


202.2 


Std. Deviation 


27.72 


27.27 


27.78 


28.68 


Confidence Interval (95%) 










Lower Bound 


200.44 


201.64 


193.28 


199.15 


Upper Bound 


203.88 


203.94 


195.63 


201.57 




Conclusion 



This study illustrated that the rater variation aeross year eould be signifieant and 
that sueh variation warrants the need for making statistieal adjustment. The polytomous 
IRT-based linking method proposed by Tate (1999, 2000) provides a way to adjust rater 
effeet in the equating proeess under the framework of non-equivalent group eommon- 
item equating design. This method was eompared with two alternative methods; 
traditional IRT linking study that links the test forms using a) all the eommon items 
without the rater adjustment and b) the eommon selected response items. The differences 
in alignment between the anchor and actual p-values were compared among the three 
methods. The differences in the test characteristic curves and the students’ scale score 
distributions were also compared. 

Significant shifts in the parameters after rater adjustment were found for one 
(grade 8) of the three grades examined. The p-values and TCCs shifted across years 
when adjusted for rater effects. The impact of the parameter shifts and TCCs manifested 
in the changes in the proficiency classification before and after adjustment. 

The results of this study have important implications to the valid interpretation of 
the scale-based results. Though time is a constraint in operational testing programs, rater 
studies need to be more routinely done to examine the direction and the magnitude of the 
rater variation. Ideally, CR items can be incorporated into anchor sets resulting in better 
year-to-year equating in mixed item format tests as long as the rater variation can be 
adjusted through the equating process. Otherwise, year-to-year growth and rater drift 
might be inextricably confounded. In the case when it is not feasible to apply rater 
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adjustment in the equating proeess, using the eommon SR items is a better approach than 
using the combined SR and CR items without adjusting for rater effect. 

Further studies need to be conducted for year-to-year test administrations that go 
beyond two administrations and for multi-categorical proficiency classifications with 
more than two levels. Furthermore, research will need to be conducted to investigate 
these effects on multiple cut scores and various levels of proximities to these cut scores. 
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